Information processing apparatus and recording medium

ABSTRACT

There is provided An information processing apparatus including a perspective switching control unit configured to switch a perspective when playing back content acquired by a content acquisition unit to at least one of a first-person perspective and a third-person perspective, an editing unit configured to edit a part of the content, and a playback control unit configured to play back the content edited by the editing unit in the at least one of the first-person perspective and the third-person perspective to which the perspective has been switched by the perspective switching control unit.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of Japanese Priority PatentApplication JP 2012-285642 filed Dec. 27, 2012, the entire contents ofwhich are incorporated herein by reference.

BACKGROUND

The present disclosure relates to an information processing apparatusand a recording medium.

The recent dramatic developments in technologies such as the mobileterminal communication speed and storage capacity and display screenprecision has enabled video content such as movies and television dramasto be easily downloaded and watched on a mobile terminal. Examples oftechnologies relating to the management of such video content includethe following.

For example, JP 2002-325241A proposes utilizing high picturequality/high audio quality movie and television program data produced byprofessionals by creating a database of movies and television programsthat have already been released or broadcast. More specifically, JP2002-325241A describes a download system that allows a user to accessand download a given part of audio data and moving image data in a videowork, which the user can utilize as a standby screen, a ringtone or thelike in a mobile terminal.

Further, JP 2007-528056T discloses a technology that automaticallyincludes link data associated with one or more pieces of content thatare associated with scene content data. It is noted that JP 2007-528056Talso describes linking the scene content data (the captured images) withGPS position information (imaging location information).

SUMMARY

However, JP 2002-325241A and JP 2007-528056T only discuss the provisionof data. Neither document mentions anything about switching perspectivewhen playing back image content.

According to an embodiment of the present disclosure, there is provideda novel and improved information processing apparatus and a recordingmedium that are capable of switching the perspective when playing backimage content to at least either a first-person perspective or athird-person perspective.

According to an embodiment of the present disclosure, there is providedan information processing apparatus including a perspective switchingcontrol unit configured to switch a perspective when playing backcontent acquired by a content acquisition unit to at least one of afirst-person perspective and a third-person perspective, an editing unitconfigured to edit a part of the content, and a playback control unitconfigured to play back the content edited by the editing unit in the atleast one of the first-person perspective and the third-personperspective to which the perspective has been switched by theperspective switching control unit.

According to an embodiment of the present disclosure, there is provideda non-transitory computer-readable recording medium having a programrecorded thereon, the program causing a computer to execute switching aperspective when playing back content acquired by a content acquisitionunit to at least one of a first-person perspective and a third-personperspective, editing a part of the content based on the switching of theperspective, and playing back the edited content in the at least one ofthe first-person perspective and the third-person perspective to whichthe switched perspective has been switched.

According to one or more of embodiments of the present disclosure,provided are a novel and improved information processing apparatus and arecording medium that are capable of switching the perspective whenplaying back image content to at least either a first-person perspectiveor a third-person perspective.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram illustrating an outline of an information processingapparatus according to an embodiment of the present disclosure;

FIG. 2 is a diagram illustrating editing processing of content by aninformation processing apparatus according to an embodiment of thepresent disclosure;

FIG. 3 is a diagram illustrating an external configuration of an HMDaccording to a first embodiment of the present disclosure;

FIG. 4 is a block diagram illustrating an internal configuration of anHMD according to a first embodiment of the present disclosure;

FIG. 5 is a diagram illustrating processing for acquiring a capturedimage of a user taken from in front by an HMD according to a firstembodiment of the present disclosure;

FIG. 6 is a diagram illustrating gesture recognition by a recognitionunit according to a first embodiment of the present disclosure;

FIG. 7 is a flowchart illustrating operation of a control unit accordingto a first embodiment of the present disclosure;

FIG. 8 is a flowchart illustrating operation of a control unit accordingto a first embodiment of the present disclosure;

FIG. 9 is a flowchart illustrating operation of a control unit accordingto a first embodiment of the present disclosure;

FIG. 10 is a diagram illustrating switching of a perspective by aperspective switching control unit according to a first embodiment ofthe present disclosure;

FIG. 11 is a diagram illustrating operation processing of an HMDaccording to a first modified example of the present disclosure;

FIG. 12 is a diagram illustrating an outline of a smartphone accordingto a second embodiment of the present disclosure;

FIG. 13 is a flowchart illustrating operation of a control unitaccording to a second embodiment of the present disclosure;

FIG. 14 is a flowchart illustrating operation of a control unitaccording to a second embodiment of the present disclosure; and

FIG. 15 is a diagram illustrating operation of a smartphone according toa second modified example of the present disclosure.

DETAILED DESCRIPTION OF THE EMBODIMENT(S)

Hereinafter, preferred embodiments of the present disclosure will bedescribed in detail with reference to the appended drawings. Note that,in this specification and the appended drawings, structural elementsthat have substantially the same function and structure are denoted withthe same reference numerals, and repeated explanation of thesestructural elements is omitted.

It is noted that the description will be made in the following order.

1. Outline of the information processing apparatus according to theembodiments of the present disclosure

-   -   2. Embodiments of the present disclosure    -   2-1. First embodiment of the present disclosure    -   2-1-1. HMD external configuration    -   2-1-2. HMD internal configuration    -   2-1-3. Operation processing    -   2-1-4. First modified example of the present disclosure    -   2-2. Second embodiment of the present disclosure    -   2-2-1. Outline    -   2-2-2. Operation processing    -   2-2-3. Second modified example of the present disclosure

3. Conclusion

1. OUTLINE OF THE INFORMATION PROCESSING APPARATUS ACCORDING TO THEEMBODIMENTS OF THE PRESENT DISCLOSURE

An outline of the information processing apparatus according to theembodiments of the present disclosure will now be described withreference to FIGS. 1 and 2. FIG. 1 is a diagram illustrating an outlineof an information processing apparatus according to an embodiment of thepresent disclosure. As illustrated in FIG. 1, the information processingapparatus according to the present embodiment is realized by aneyeglasses-type HMD (head mounted display) 1. As described in detailbelow, the HMD 1 is an apparatus that is worn by a user 2 for playingback content to the user 2. The HMD 1 can build a realistic virtualreality by projecting images over the entire field of view of the user2. This content includes at least moving image-based or stillimage-based images (image data). For example, the content may be amovie, a television drama, animation or the like. In the presentspecification, the description will be based on a movie as the content.

Generally, when viewing content, the user 2 views from the perspectiveof the camera that captured that content (a third-person perspective).Namely, the user 2 views the content as if he/she was present at thelocation when the content was captured and looking at the imaging target(object) from the position of the camera lens. Examples of the imagingtarget include a living being, such as an actor, a dog, or a catappearing in a movie or a television drama, a virtual living beingrealized by a robot or a CG (computer graphic), an animation characterand the like.

Here, a case will be described in which the user 2 views content 101captured at a scene 100 illustrated in FIG. 1. As illustrated in FIG. 1,at this stage, the user 2 views the content 101 as if looking at theimaging targets in the scene 100 (the actress 3 and actor 4 who make anappearance in content 101) from the position of the camera that capturedthe content 101. It is noted that when the content 101 was captured, theactress 3 and actor 4 were at the position of the origin where theX-axis, Y-axis, and Z-axis of the scene 100 illustrated in FIG. 1intersect. Further, the camera that captured the content 101 ispositioned on the X-axis illustrated in FIG. 1 during image capture.

Here, as described above, JP 2002-325241A and JP 2007-528056T onlydisclose an information processing apparatus that provides imagecontent, they do not contain any description about switching perspectivewhen playing back image content.

However, if the perspective when playing back the image content can beswitched from a simple third-person perspective to a first-personperspective, which is the perspective as seen from an imaging targetappearing in the image content, the entertainment value of the imagecontent is improved.

Accordingly, the information processing apparatuses according to therespective embodiments of the present disclosure were created byfocusing on the above-described situation. The information processingapparatuses according to the respective embodiments of the presentdisclosure can play back content by switching the perspective whenplaying back content to at least either the first-person perspective orthe third-person perspective.

For example, as illustrated in FIG. 1, the HMD 1 can switch theperspective from the above-described third-person perspective to afirst-person perspective, which is the perspective as seen from theactor 4 appearing in the content 101, and play back content 104 in thefirst-person perspective. The content 104 is image content in which theactress 3 appears as seen from the perspective of the actor 4.Consequently, the user 2 is immersed in the world of the content, andcan enjoy the content as if he/she were actually playing the role of theactor 4 himself/herself.

Here, even within the first-person perspective, the HMD 1 can furtherswitch perspective and play back the content. For example, asillustrated in FIG. 1, the HMD 1 can switch from the position of theactor 4 to a perspective even closer to the actress 3, and play backcontent 105. Consequently, the user 2 can enjoy the content as if he/shewere playing the role of the actor 4 at a position nearer to the actress3. In addition, the HMD 1 can switch to an even closer perspective, andplay back content 106 and content 107.

On the other hand, even within the third-person perspective, the HMD 1can further switch perspective and play back the content. For example,the HMD 1 can switch to a perspective 2-2, which is a position differentto the camera that captured the content 101, and play back content 102.In this case, the user 2 can view the content 102 as if looking at theactress 3 and the actor 4 in the scene 100 from the position ofperspective 2-2. Further, the HMD 1 can switch to perspective 2-3, andplay back content 103.

Moreover, the HMD 1 according to the present embodiment can even edit apart of the content. Content editing by the HMD 1 will be describedbelow with reference to FIG. 2.

FIG. 2 is a diagram illustrating editing processing of content by aninformation processing apparatus according to an embodiment of thepresent disclosure. As illustrated in FIG. 2, in content 101A, which iscontent as seen from a third-person perspective, the actress 3 and theactor 4 are facing each other. The HMD 1 according to the presentembodiment can play back content 101B that has been edited as if theface portion of the actor 4 appearing in the content 101A was replacedwith the face of the user 2. Consequently, the user 2 can enjoy thecontent as if he/she were viewing content in which the userhimself/herself appeared.

Here, when switching to a first-person perspective after having seencontent in a third-person perspective in which his/her own profileappears, the user 2 can experience a feeling of returning to his/her ownbody that until now he/she had been looking at from an externalperspective. Further, when switching from a first-person perspective toa third-person perspective in which he/she sees content in which his/herown profile appears, the user 2 can see his/her own body from outside,namely, experience a feeling of having left his/her body. Combined withthe ability of the HMD 1 to build a realistic virtual reality, thesefeelings are felt as if he/she has actually left his/her body.

Thus, the user 2 can become immersed in and enjoy the content by lookingat content in which a part has been replaced with his/her own profile.

In the above, an outline of the information processing apparatusaccording to an embodiment of the present disclosure was described.Next, the respective embodiments of the present disclosure will bedescribed in turn. It is noted that in the example illustrated in FIG.1, although an eyeglasses-type HMD 1 was used as an example of theinformation processing apparatus, the information processing apparatusaccording to an embodiment of the present disclosure is not limited tothis. The information processing apparatus may be some other apparatus,such as, for example, a digital camera, a digital video camera, a PDA(personal digital assistant), a notebook PC, a tablet terminal, a mobiletelephone terminal, a smartphone, a portable music player, a portableimage processing device, or a mobile game device.

2. EMBODIMENTS OF THE PRESENT DISCLOSURE 2-1. First Embodiment of thePresent Disclosure 2-1-1. HMD External Configuration

First, the external configuration of the HMD 1 illustrated in FIG. 1will be described with reference to FIG. 3, and then the internalconfiguration will be described with reference to FIGS. 4 to 6.

FIG. 3 is a diagram illustrating an external configuration of an HMDaccording to the first embodiment of the present disclosure. The HMD 1illustrated in FIG. 3 is termed an eyeglasses-type display or asee-through HMD. Specifically, the HMD 1 has a mount unit with a framestructure that extends from either side of the head half-way around tothe rear of the head, and is mounted on the user 2 by placing over bothears as illustrated in FIG. 3. In a mounting state like that illustratedin FIG. 3, this HMD 1 includes a pair of display units 60 a and 60 b forthe left and right eyes, respectively, immediately in front of eithereye of the user 2, namely, at a location where the lens in ordinaryeyeglasses would be. A liquid crystal panel, for example, may be usedfor the display unit 60 (display unit 60 a and display unit 60 b). TheHMD 1 can be set to a see-through state like that illustrated in FIG. 3,namely, a transparent or semi-transparent state, by controlling thetransmittance of the liquid crystal panel. By setting the display unit60 to a see-through state, there are no hindrances to normal activityeven if the user 2 constantly wears the HMD 1 like eyeglasses.

In addition, the display unit 60 can superimpose AR (augmented reality)information on the actual spatial landscape by displaying an image oftext, a graphic or the like in the transparent or semi-transparentstate.

Moreover, the display unit 60 can display a captured image of the actualspace captured by imaging lenses 20 a and 20 b, and superimpose AR(augmented reality) information over that captured image of the actualspace. Still further, the display unit 60 can play back and displaycontent received by the HMD 1 from an external device or content storedon a storage medium (the below-described storage unit 40) of the HMD 1.Examples of the external device may include a server, a digital camera,a digital video camera, and an information processing apparatus such asa mobile telephone terminal, a smartphone, and a personal computer.

Examples of the content played back and displayed on the display unit 60include moving image content, such as a movie or a video clip, stillimage content captured by a digital still camera and the like, and datain an electronic book, for example. Further, this data may be any datathat can be displayed, such as computer-use data like image data, textdata, and spreadsheet data created by the user 2 on a personal computeror the like, and game images derived from a game program.

In addition, although only displayed on the right side of FIG. 3, theimaging lenses 20 a and 20 b are arranged facing the user 2 so as tocapture an image of the user 2 from close range in a state in which theuser 2 is wearing the HMD 1. Here, the imaging lens 20 a captures animage of the upper right portion of the user 2's face, and the imaginglens 20 b captures an image of the lower right portion of the user 2'sface and the upper right half of the user 2's body. Similarly, althoughnot illustrated, an image of the upper left portion of the user 2's faceas well as an image of the lower left portion of the user 2's face andthe upper left half of the user 2's body are captured with imaginglenses (not illustrated) provided on the left eye side. As describedbelow, the HMD 1 can generate a captured image that looks like it wastaken from in front of the user 2 by combining these captured images ofthe user 2 taken from close range.

Further, imaging lenses 20 c and 20 d are arranged facing forward so asto capture the direction visible to the user 2 as a subject direction ina state in which the user 2 is wearing the HMD 1.

In addition, although only displayed on the left ear side in FIG. 3, apair of earphone speakers 70 a is provided that can be inserted into theleft and right ear holes of the user 2 when worn.

Moreover, microphones 30 a and 30 b for collecting external audio arearranged to the right of the display unit 60 a for the right eye and tothe left of the display unit 60 b for the left eye, respectively.

It is noted that the external configuration of the HMD 1 illustrated inFIG. 3 is an example. Various other configurations may be employed formounting the HMD 1 on the user 2. It is usually sufficient for the HMD 1to be formed from an eyeglasses-type or head-mounted type mounting unit.At the least, in the present embodiment it is sufficient for the displayunit 60 to be provided close to the front of the user 2's eyes. Further,the display unit 60 may also be provided as a single unit correspondingto the eye on one side, rather than as a pair of units corresponding toeach eye.

In addition, the earphone speaker 70 a may be provided as a single unitfor wearing on just one ear, rather than providing left and rightearphone speakers. Moreover, the microphone may also be provided as justthe microphone 30 a or the microphone 30 b.

Still further, the HMD 1 may even be configured without including themicrophones 30 a and 30 b and the earphone speaker 70 a.

In the above, the external configuration of the HMD 1 according to anembodiment of the present disclosure was described.

2-1-2. HMD Internal Configuration

FIG. 4 is a block diagram illustrating an internal configuration of theHMD 1 according to a first embodiment of the present disclosure. Asillustrated in FIG. 4, the HMD 1 includes a control unit 10, an imagingunit 20, an audio input unit 30, a storage unit 40, a communication unit50, a display unit 60, and an audio output unit 70.

(Imaging Unit)

The imaging unit 20 includes, for example, a lens system configured fromthe imaging lenses 20 a, 20 b, 20 c, 20 d, a diaphragm, a zoom lens, anda focus lens, a drive system that makes the lens system perform a focusoperation and a zoom operation, and a solid-state image sensor arraythat generates an imaging signal by photoelectrically convertingcaptured light obtained by the lens system. The solid-state image sensorarray may be realized by, for example, a CCD (charge-coupled device)sensor array or a CMOS (complementary metal oxide semiconductor) sensorarray.

Further, in the present embodiment, as described above with reference toFIG. 3, the lenses 20 a and 20 b are arranged so as to capture an imageof the user 2 from close range in a state in which the HMD 1 is worn bythe user. Therefore, the lenses 20 a and 20 b can capture close-upimages of the user 2 from close range and from various angles. Thesecaptured images of the user 2 taken from close range are combined by thebelow-described editing unit 12 to generate a captured image that lookslike it was taken from in front of the user 2.

In addition, in the present embodiment, as described above withreference to FIG. 3, the lenses 20 c and 20 d are arranged facingforward so as to capture the direction visible to the user 2 as asubject direction in a state in which the user 2 is wearing the HMD 1.Therefore, the lenses 20 c and 20 d can capture an image of a range thatincludes the range (field of view) that can be seen by the user 2through the display unit 60. A gesture made by the user 2 is recognizedby the below-described recognition unit 13 based on these capturedimages that capture the direction visible to the user 2.

The imaging unit 20 outputs the captured images to the control unit 10.

(Audio Input Unit)

The audio input unit 30 includes the microphones 30 a and 30 billustrated in FIG. 3, and a micro amplifier unit and an A/D converterthat amplify and process the audio signals obtained by the microphones30 a and 30 b. The audio input unit 30 outputs the resultant audio datato the control unit 10.

In the present embodiment, the audio input unit 30 collects the voice ofthe user 2 wearing the HMD 1. The voice of the imaging target includedin the content may be replaced by the below-described editing unit 12with the voice of the user 2 collected by the audio input unit 30.

(Storage Unit)

The storage unit 40 is a unit for performing recording and playback ofdata in a predetermined storage medium. The storage unit 40 is realizedas a HDD (hard disk drive), for example. Obviously, as long as thestorage unit 40 can execute recording and playback on the recordingmedium that is employed, various other kinds of medium may also be usedas the recording medium, such as a memory card, an optical disc, amagneto-optical disk, a hologram memory and the like that include afixed memory.

In the present embodiment, the storage unit 40 stores, for example,moving image content, such as a movie or a video clip and still imagecontent captured by a digital still camera and the like associated withposition information. More specifically, the storage unit 40 storesimage content, photograph content, text content and the like that isassociated with the location where that movie or photograph was taken orthe location where that content was set (modeled).

For example, the storage unit 40 associates and stores famous situationsand famous scenes from various pieces of image content (movies,television dramas etc.) with position information, a name, an image andthe like that specifies the location where that scene was captured. Itis noted that position information that specifies a location may belatitude and longitude, for example. Further, a name that specifies alocation may be an address, a place name, a facility name, or a buildingname, for example. In addition, an image that specifies a location maybe a captured image of that location, or a captured image of acharacteristic building or landscape in the vicinity of that location.

Moreover, a title of the image content that includes that scene, a titlescreen, a main theme song or the like may also be associated with eachscene.

Further, the content may also be associated with information indicatingwhether that content is based on a first-person perspective or based ona third-person perspective. Moreover, the content may be associated withinformation indicating whose (which actor) the first-person perspectivethat content is based on.

(Communication Unit)

The communication unit 50 sends and receives data to and from anexternal device. The communication unit 50 is an example of a unit foracquiring external information. The communication unit 50 may performnetwork communication via near field communication with a network accesspoint using wireless LAN (local area network), Bluetooth® or the like,for example. Further, the communication unit 50 may also perform nearfield communication with an external device that includes a compatiblecommunication function.

This external device may be any device that includes an informationprocessing and a communication function, such as a computer device, aPDA, a mobile telephone terminal, a smartphone, a video device, an audiodevice, a tuner device and the like. Further examples of the externaldevice to which communication is performed include a terminal apparatusor a server apparatus connected to a network such as the Internet. Inaddition, the external device may be a two-dimensional bar code, such asa non-contact communication IC card or a QR code (registered trademark)that includes an IC chip, or a hologram memory, from which thecommunication unit 50 reads information. In addition, another HMD 1 mayalso serve as the external apparatus.

In the present embodiment, the communication unit 50 receives, forexample, moving image content, such as a movie or a video clip, or stillimage content captured by a digital still camera and the like, andoutputs the received content to the control unit 10.

(Audio Output Unit)

The audio output unit 70 includes a pair of the earphone speakers 70 aillustrated in FIG. 3 and an amplifier circuit for the earphone speakers70 a. Further, the audio output unit 70 may be formed as a so-calledbone-conduction speaker. The audio output unit 70 allows the user tolisten to external audio, audio played back by the content stored in thestorage unit 40, and audio received by the communication unit 50.

(Control Unit)

The control unit 10 is configured from, for example, a microcomputerthat includes a CPU (central processing unit), a ROM (read-only memory),a RAM (random-access memory), a non-volatile memory, and an interfaceunit. The control unit 10 controls each of the units in the HMD 1.

Further, as illustrated in FIG. 4, the control unit 10 functions as acontent acquisition unit 11, an editing unit 12, a recognition unit 13,a position information acquisition unit 14, a perspective switchingcontrol unit 15, and a playback control unit 16.

Content Acquisition Unit

The content acquisition unit 11 acquires as content moving image-basedor still image-based image data and audio data output from the imagingunit 20, the audio input unit 30, the storage unit 40, or thecommunication unit 50.

Especially, the content acquisition unit 11 according to the presentembodiment acquires content that is associated with position informationindicating the current position of the HMD 1 that is acquired by theposition information acquisition unit 14. For example, if an imagecaptured in the past at a position where the HMD 1 is currently presentis stored in the storage unit 40, the content acquisition unit 11acquires a scene from the movie captured at that position as content. Inaddition, if content specified by a building or a landscape that wascaptured by the imaging unit 20 (imaging lenses 20 c and 20 d) and isincluded in the field of view of the user 2 is stored in an externaldevice, the content acquisition unit 11 can acquire that content via thecommunication unit 50. For example, the content acquisition unit 11acquires a scene from a television drama captured on a road as contentby making an inquiry to an external device via the communication unit 50about whether the external device has a television drama that wascaptured in the past at the road in front of the user 2.

It is noted that the content acquisition unit 11 according to thepresent embodiment is not limited to acquiring the above-describedcontent that is associated with position information, the contentacquisition unit 11 may acquire content specified by the user 2. Forexample, the content acquisition unit 11 can acquire content about atitle input by the user 2, or automatically acquire content in which asong that the user 2 is currently listening to is set as a theme song.

Further, the content acquisition unit 11 acquires and generates in realtime a below-described morphing image from the storage unit 40 or anexternal device connected via the communication unit 50.

The content acquisition unit 11 outputs the acquired content to theediting unit 12, the recognition unit 13, and the playback control unit16.

Editing Unit

The editing unit 12 edits a part of the content played back by theplayback control unit 16. More specifically, the editing unit 12performs editing by replacing image data of an imaging target appearingin the played-back content with image data of another imaging targetappearing in some other content acquired by the content acquisition unit11. For example, as described above with reference to FIG. 2, theediting unit 12 can replace image data of the face portion of the actor4 appearing in the content with image data of the face portion of theuser 2 appearing in another captured image.

Here, to perform this replacement of image data, the editing unit 12 mayapply the technology disclosed in JP 2010-86178A. This technologycaptures and registers in advance face images of the user 2's face fromdifferent orientations and in various emotions, and replaces an actor'sface with a CG (computer graphic) by selecting the face image with thehighest correlation to the orientation and expression of the face of theactor appearing in the content from among these registered face imagesof the user. By applying this technology, the editing unit 12 canreplace the face of the actor appearing in the content with the faceimage having a high correlation among the face images of the user thatwere registered in advance in the storage unit 40 or an external device.

Further, in addition to using pre-registered face images, the editingunit 12 can use a face image of the user 2 acquired in real time.However, it is difficult for the HMD 1 to acquire a captured image takenfrom in front of the user 2, for example, while the HMD 1 is being wornby the user 2. Accordingly, the HMD 1 according to the presentembodiment acquires in real time a captured image taken from in front ofthe user 2 based on the following method described with reference toFIG. 5 using the imaging unit 20 (imaging lenses 20 a and 20 b) arrangedso as to capture an image of the user 2 from close range.

FIG. 5 is a diagram illustrating processing for acquiring a capturedimage of the user taken from in front by the HMD 1 according to thepresent embodiment. As illustrated in FIG. 5, a captured image 111 is acaptured image of the upper right portion of the user 2's face capturedby the imaging lens 20 a. A captured image 112 is a captured image ofthe lower right portion of the user 2's face and the upper right half ofthe user 2's body captured by the imaging lens 20 b. A captured image113 is a captured image of the upper left portion of the user 2's facecaptured by an imaging lens (not illustrated) provided on the left sideeye. A captured image 114 is a captured image of the lower left portionof the user 2's face and the upper left half of the user 2's bodycaptured by an imaging lens (not illustrated) provided on the right sideeye. By combining these captured images 111, 112, 113, and 114, theimage processing apparatus according to the present embodiment acquiresa captured image 115 in which the upper half, including the face, of theuser 2's body appears as if taken from in front. Further, the imageprocessing apparatus according to the present embodiment can replace theface image of a front-facing actor captured in the content with thefrontal face image of the user 2 using the captured image acquired bythe above-described method.

In addition, the image processing apparatus according to the presentembodiment can also similarly generate a captured image of the user 2that looks as if it was taken from the side or from the rear byarranging the imaging unit 20 so as capture an image of the user 2 fromthe side of the head or the rear of the head. Therefore, the imageprocessing apparatus according to the present embodiment can replacewith the current side-on face profile or the rear profile of the user 2even when the actor 4 captured in the content is not facing toward thefront. This enables the HMD 1 to avoid the need to capture and registerin advance face images of the user 2's face from different orientationsand in various emotions. Further, the user 2 can enjoy in real timecontent that reflects his/her current profile.

Still further, a case will now be considered in which the imageprocessing apparatus according to the present embodiment replaced therear head portion of the actor appearing in the content with the currentrear head portion of the user 2. In this case, when switching from afirst-person perspective to a third-person perspective, since the user 2can see his/her own rear head portion while leaving his/her own body,the user 2 can more strongly experience the feeling of having lefthis/her body. Further, when switching from a first-person perspective toa third-person perspective, the user 2 can more strongly experience thefeeling of returning to his/her own body from a state in which he/shehad left it, by gradually incorporating his/her own rear head portion.

Moreover, although a case was described above in which the imageprocessing apparatus according to the present embodiment replaces a partof the content with the user 2 who is wearing the HMD 1, the presentembodiment is not limited to this example. For example, face images ofanother person's face from different orientations and in variousemotions may be captured and registered in advance, and the editing unit12 may replace an actor's face with a CG by selecting the face imagewith the highest correlation to the orientation and expression of theface of the actor appearing in the content from among these registeredface images of the other person. In addition, the editing unit 12 mayalso replace a part of the content with the other person's profile usinga captured image of the other person acquired in real time by theimaging unit 20 (imaging lenses 20 c and 20 d) that is arranged so as tocapture the direction visible to the user 2 as the subject direction.

It is noted that when replacing the face image, portions where color isnot continuous can occur at a boundary portion between the replacedimage and the surrounding images. In this case, the editing unit 12 mayapply the technology disclosed in JP 2002-269546A. This is a technologyin which the pixel values of the pixels at a portion where the replacedimage and a surrounding image overlap are mixed so that the color at theboundary portion changes naturally. By applying this technology, theediting unit 12 can correct the portions where color is not continuousthat occur at a boundary portion between the replaced image and thesurrounding images.

Further, the editing unit 12 can also replace the audio data of theimaging target appearing in the content with other audio data appearingin other content acquired by the content acquisition unit 11. Forexample, the editing unit 12 replaces the audio data of the actor 4appearing in the content with audio data of the user 2 acquired by theaudio input unit 30.

The editing unit 12 performs the above-described editing processing onthe content output from the playback control unit 16, and outputs theedited content to the playback control unit 16.

Recognition Unit

The recognition unit 13 recognizes an instruction to switch perspective.For example, the recognition unit 13 recognizes an instruction to switchperspective by recognizing a captured image output from the imaging unit20 in which a gesture by the user 2 appears, recognizing the voice ofthe user 2 output from the audio output unit 30, or recognizing anot-illustrated button/switch operation. In the present specification,the recognition unit 13 recognizes an instruction to switch perspectivesbased on a gesture by the user 2.

The recognition unit 13 can recognize various gestures made using theuser 2's body and hands. An example will be described below withreference to FIG. 6 in which a gesture made with the fingers isrecognized. FIG. 6 is a diagram illustrating gesture recognition by therecognition unit 13 according to the first embodiment of the presentdisclosure. It is noted that the recognition unit 13 recognizes agesture made by fingers when the user 2 waves his/her in front ofhimself/herself from a captured image output from the imaging unit 20(imaging lenses 20 c and 20 d) that captured the front of the user 2.

The gesture 121 illustrated in FIG. 6 illustrates an operation in whichtwo fingers are spread. If the recognition unit 13 recognizes thegesture 121 made with the fingers of the user 2, the recognition unit 13recognizes that the user 2 is issuing an instruction to switchperspectives from a third-person perspective to a first-personperspective. Here, switching from a third-person perspective to afirst-person perspective means that an imaging target that was seen faraway is now seen close up. This is similar to an image being enlarged ifa similar operation is performed on a touch panel when operating asmartphone or the like. Therefore, the user 2 can issue an instructionto switch perspectives based on a similar familiar operation to thatperformed when operating a smartphone or the like.

On the other hand, the gesture 122 illustrated in FIG. 6 illustrates anoperation in which two fingers are brought closer together. If therecognition unit 13 recognizes the gesture 122 made with the fingers ofthe user 2, the recognition unit 13 recognizes that the user 2 isissuing an instruction to switch perspectives from a first-personperspective to a third-person perspective. Here, switching from afirst-person perspective to a third-person perspective means that animaging target that was seen up close is now seen far away. This issimilar to an image being made smaller if a similar operation isperformed on a touch panel when operating a smartphone or the like.Therefore, the user 2 can issue an instruction to switch perspectivesbased on a similar familiar operation to that performed when operating asmartphone or the like.

Further, the recognition unit 13 can also simultaneously recognizeinformation indicating which actor the user 2 wants to become whenrecognizing the instruction to switch from a third-person perspective toa first-person perspective. For example, if the user 2 has performed agesture with his/her fingers so that they are superimposed over an actorappearing in semi-transparently displayed content on the display unit60, the recognition unit 13 can recognize this gesture as an instructionto switch to the first-person perspective content as seen from thatactor.

The recognition unit 13 outputs the recognized switch instruction madeby the user 2 to the perspective switching control unit 15.

Position Information Acquisition Unit

The position information acquisition unit 14 acquires positioninformation about the HMD 1. For example, the position informationacquisition unit 14 receives radio waves from a GPS (global positioningsystem) satellite, and acquires position information includinglatitude/longitude/altitude indicating the current position of the HMD1. Further, when the user 2 uses the HMD 1 in a confined space, such asindoors, the position information acquisition unit 14 may acquire theposition information based on an indoor positioning technology.

Various indoor positioning technologies may be utilized by the HMD 1.For example, the position of the HMD 1 can be measured based on beaconinformation from about a plurality of wireless access points arrangedindoors. Further, the position information acquisition unit 14 can alsomeasure the position of the HMD 1 based on position information about anIMES (Indoor MEssaging System) transmitter transmitted from an IMEStransmitter arranged indoors. Still further, the position informationacquisition unit 14 can measure the position of the HMD 1 by, afterhaving acquired a start point position with a given device, calculatingthe current position in combination with an acceleration sensor or agyro sensor.

In addition to the latitude/longitude/altitude indicating the currentposition of the HMD 1, the position information acquisition unit 14 canacquire the bearing and the elevation of the HMD 1. For example, theposition information acquisition unit 14 acquires the bearing of the HMD1 with an electronic compass. Here, the bearing of the HMD 1 indicatesthe orientation of the user 2's face. Further, the position informationacquisition unit 14 acquires the elevation of the HMD 1 with a gyrosensor. Here, the elevation of the HMD 1 indicates the elevation of theuser 2's face.

Moreover, the position information acquisition unit 14 may also acquirea relative position with respect to a specific position as the positioninformation about the HMD 1. For example, when using a smartphone 6indoors, a specific position indoors is pre-set as a position where theimaging target was in the imaging location, and a relative position withrespect to that specific position is acquired as the positioninformation.

The position information acquisition unit 14 outputs the acquiredposition information to the content acquisition unit 11 and theperspective switching control unit 15.

Perspective Switching Control Unit

The perspective switching control unit 15 switches the perspective whenplaying back the content acquired by the content acquisition unit 11with the display unit 60 to at least one of either a first-personperspective or a third-person perspective. The perspective switchingcontrol unit 15 switches the perspective based on the switch instructionindicated by the gesture made by the user 2 that was recognized by therecognition unit 13. In addition, the perspective switching control unit15 may also switch to a perspective from a position corresponding to theposition information acquired by the position information acquisitionunit 14 in the space of the content. It is noted that an example of theperspective switching control unit 15 switching perspective based onposition information will be described in the below second embodiment ofthe present disclosure.

Playback Control Unit

The playback control unit 16 plays back the content edited by theediting unit 12 in the perspective switched to by the perspectiveswitching control unit 15. More specifically, the playback control unit16 controls the display unit 60 and the audio output unit 70 so that thecontent that was partially edited by the editing unit 12 is played backfrom at least either a first-person perspective or a third-personperspective. Consequently, if there has been a perspective switch by theperspective switching control unit 15, the playback control unit 16again acquires the content as seen from the switched-to perspective fromthe content acquisition unit 11, and plays back the acquired content.

For example, the playback control unit 16 plays back the normal contentwhen playing back content from a third-person perspective, and whenplaying back content from a first-person perspective, acquires and playsback the movie content from an actor's perspective that has been createdin advance. The content from a first-person perspective may be moviecontent from an actor's perspective created in advance by mounting acamera on the actor's head when capturing the content and having himperform the same acting as the real performance.

In addition, for example, by applying the free-perspective imagegeneration technology disclosed in JP 2006-211531A, the playback controlunit 16 can generate in real time content that is based on a perspectivefrom a given position in the space of the content with the contentacquisition unit 11. In free-perspective image generation technology,images from a given perspective are generated by merging images obtainedfrom a plurality of cameras based on information about the cameraposition and imaging angle when capturing the images, and a framingpattern, for example, and performing three-dimensional analysis on themerged image. By applying this technology, the playback control unit 16can play back content from a given position in a third-personperspective, and play back content from where an actor is in afirst-person perspective.

The playback control unit 16 edits the thus-switched content in theediting unit 12, and plays back the edited content with the display unit60 and the audio output unit 70. The playback control unit 16 alsoperforms the following below-described processing when playing back thecontent edited by the editing unit 12.

When the perspective is switched by the perspective switching controlunit 15, the playback control unit 16 plays back a morphing image thatgradually changes from the perspective before the perspective wasswitched to the switched perspective. Here, “morphing” refers to aspecial imaging technology based on CG that is generally used in moviesand animation, in which the morphing expresses a gradual change from agiven form to another form. The morphing image may be pre-stored in thestorage unit 40, or may be externally acquired by the communication unit50, or may be generated in real time by the content acquisition unit 11,for example, during switching.

Here, when switching the perspective, the playback control unit 16 canplay back the gradually-changing morphing image so that rather thansimply changing, the morphing image is changed using various techniquesthat are typical in camera-based imaging techniques. For example, theplayback control unit 16 can play back the gradually-changing morphingimage so that the image it looks as if it was captured using imagingtechniques such as zoom in, zoom out, dolly in, dolly back, or panning.

Zoom in is an imaging technique in which the subject is enlarged byincreasing the magnification of the imaging lens when capturing an imagewith a camera. When filming a movie or the like, zoom in is performedfor the purpose of, for example, emphasizing a specific subject, guidingthe viewer's line of sight, increasing tension and the like. Further,zoom out is an imaging technique in which the subject is made smaller bydecreasing the magnification of the imaging lens when capturing an imagewith a camera. When filming a movie or the like, zoom out is performedfor the purpose of, for example, illustrating the surrounding situationor the positional relationship of the subject, reducing tension and thelike.

Dolly in is an imaging technique in which the subject is enlarged bybringing the camera closer to the subject when capturing an image with acamera. Further, dolly back is an imaging technique in which the subjectis made smaller by moving the camera further away from the subject whencapturing an image with a camera. When filming a movie or the like,dolly in/out is performed for the purpose of, for example, giving animpression that the viewer is acting as the main character.

Panning is an imaging technique that follows the subject's movementswhen capturing an image with a camera. Types of panning include animaging technique that follows the subject's movements by fixing thecamera position and changing only the direction that the camera ispointing, and an imaging technique that follows the subject's movementswhile moving the camera itself. When filming a movie or the like,panning is performed for the purpose of, for example, illustrating thesurrounding situation or the positional relationship of the subject,showing a horizontally wide or a vertically tall imaging target and thelike.

When switching the perspective from a third-person perspective to afirst-person perspective, the playback control unit 16 plays back amorphing image that changes so that first the perspective zooms in, andthen dollies in. The playback control unit 16 can smoothly express theswitch from a third-person perspective to a first-person perspective by,first, enlarging the subject in a third-person perspective by zoomingin, and then further enlarging the subject in a first-person perspectiveby dollying in. It is noted that when switching the perspective from athird-person perspective to a first-person perspective, the playbackcontrol unit 16 can also play back a morphing image that changes so thatfirst the perspective dollies in, and then zooms in.

Further, when switching the perspective from a first-person perspectiveto a third-person perspective, the playback control unit 16 plays back amorphing image that changes so that first the perspective dollies back,and then zooms out. The playback control unit 16 can smoothly expressthe switch from a first-person perspective to a third-person perspectiveby, first, making the subject in a first-person perspective smaller bydollying back, and then making the subject in a third-person perspectiveeven smaller by zooming out. It is noted that when switching theperspective from a first-person perspective to a third-personperspective, the playback control unit 16 can also play back a morphingimage that changes so that first the perspective zooms out, and thendollies back.

In addition, if the camera position where the pre-perspective-switchingcontent was captured and the camera position where thepost-switching-perspective content was acquired are different, amorphing image is played back that changes so that the perspective pans.Consequently, the playback control unit 16 can play back content whilegiving the user 2 an impression as if the images were captured while thecamera position is moving.

The above-described example of a morphing image that gradually changesbased on imaging techniques such as zoom in, zoom out, dolly in, dollyback, panning and the like will now be described with reference to FIG.10.

Further, the playback control unit 16 may also change the localizedposition of the audio image based on the perspective for playing backthe content. For example, when playing back the content 101 illustratedin FIG. 1 in a third-person perspective, the playback control unit 16fixes the voice of the actress 3 and the actor 4 at the far side as seenfrom the user 2. Further, when playing back the content 104 illustratedin FIG. 1 in a first-person perspective, the playback control unit 16fixes the voice of the actress 3 right in front of the user 2, and fixesthe voice of the actor 4 at the position where the user 2 ishimself/herself at. In addition, when the perspective is switched by theperspective switching control unit 15, the playback control unit 16 canplay back the content so that the localized position of the audio imagegradually changes so as to bridge the difference between these localizedpositions. For example, when switching from a third-person perspectiveto a first-person perspective, an audio image that was fixed far awayfrom the user 2 may be gradually changed so as to come closer to theuser 2. Consequently, the user 2 can feel a sense of realness due to theswitch in perspective.

Further, when the perspective is switched by the perspective switchingcontrol unit 15, the playback control unit 16 can play backpredetermined audio data as a sound effect. For example, when switchingfrom a first-person perspective to a third-person perspective, theplayback control unit 16 can induce a-waves by playing back audio datain which the difference between the frequency of the sound entering theleft ear and the frequency of the sound entering the right ear iscontrolled. Consequently, the user 2 is relaxed by the a-waves, whichallows the user 2 to experience the above-described feeling of havingleft his/her body in a more natural manner. It is noted that this audiodata may be any data relating to sound, such as voices, music, ambientsounds and the like.

In the above, the internal configuration of the HMD 1 according to thepresent embodiment was described.

2-1-3. Operation Processing

Next, the operation processing of the HMD 1 according to the presentembodiment will be described with reference to FIGS. 7 to 10. FIGS. 7 to9 are flowcharts illustrating operation of the control unit 10 accordingto the first embodiment of the present disclosure. As illustrated inFIG. 7, first, in step S10, the control unit 10 performs initializationprocessing. More specifically, the control unit 10 performsinitialization processing such as reading various setting values fromthe non-volatile memory and setting initial values.

Next, in step S20, the position information acquisition unit 14 acquiresposition information, which includes the current position, the bearing,and the elevation of the HMD 1.

Next, in step S30, the content acquisition unit 11 determines whethercontent corresponding to the position information acquired by theposition information acquisition unit 14 is present. For example, thecontent acquisition unit determines whether the storage unit 40 storescontent associated with the acquired position information. Additionally,the content acquisition unit 11 may also make an inquiry to an externaldevice via the communication unit 50 about the presence of contentassociated with the acquired position information.

If content corresponding to the position information is not present (NOin S30), the processing returns to step S20.

On the other hand, if content corresponding to the position informationis present (YES in S30), in step S40, the control unit 10 performscontent display processing. It is noted that the content displayprocessing will be described in more detail below.

Next, in step S50, the control unit 10 determines whether to finish theprocessing. For example, the control unit 10 determines to finish theprocessing if a finish button has been pressed, if the content has beenplayed back a predetermined number of times in step S30, or if apredetermined duration has elasped without any operation being made.

If it is determined to not finish the processing (NO in S50), theprocessing returns to step S40.

On the other hand, if it is determined to finish the processing (YES inS50), in step S60, the control unit 10 performs finishing processing.For example, the control unit 10 performs finishing processing such aswriting in the non-volatile memory history information indicating whichcontent was played where and how many times, and the various settingvalues changed by the user 2 during display.

Next, the above-mentioned content display processing performed in stepS40 will be described with reference to FIGS. 8 and 9. As illustrated inFIG. 8, first, in step S102, the perspective switching control unit 15determines whether the perspective is a third-person perspective. Forexample, the perspective switching control unit 15 determines whetherthe perspective is a third-person perspective by referring toinformation indicating whether the content associated with the contentto be played back is content based on a first-person perspective or on athird-person perspective.

If the perspective is not a third-person perspective (NO in S102), theprocessing proceeds to the below-described step S204.

On the other hand, if the perspective is a third-person perspective (YESin S102), in step S104, the editing unit 12 performs editing processingon the content that was acquired by the content acquisition unit 11 in athird-person perspective. More specifically, the editing unit 12, first,as described with reference to FIG. 5, acquires a captured image thatcaptures the current profile of the user 2. Then, the editing unit 12replaces the face image of the imaging target, such as an actor or acharacter appearing in the content, with the face image of the user 2 inthe manner described above with reference to FIG. 2.

Further, in step S108, the playback control unit 16 controls so that thecontent edited by the editing unit 12 is displayed by the display unit60 and the audio output unit 70. The content that is played back at thistime will be described with reference to FIG. 10.

FIG. 10 is a diagram illustrating switching of the perspective by theperspective switching control unit 15 according to the first embodimentof the present disclosure. As illustrated in FIG. 10, the playbackcontrol unit 16 plays back content 101B based on the third-personperspective set by the perspective switching control unit 15. Asillustrated in FIG. 10, the face image of the actor 4 appearing in thecontent 101B is replaced with the face image of the user 2 by theediting unit 12.

Next, in step S112, the control unit 10 determines whether to stopplayback. For example, the control unit 10 determines to stop playbackif a playback stop button has been pressed.

If it is determined to stop playback (YES in S112), the content displayprocessing performed by the control unit 10 is finished.

On the other hand, if it is determined not to stop playback (NO inS112), in step S116, the recognition unit 13 determines whether aninstruction by the user 2 to switch from a third-person perspective to afirst-person perspective has been recognized. For example, therecognition unit 13 determines whether a button/switch operationinstructing a switch has been made or whether the gesture describedabove with reference to FIG. 6 has been recognized.

If an instruction to switch from a third-person perspective to afirst-person perspective is not recognized (NO in S116), the processingreturns to step S104.

If an instruction to switch from a third-person perspective to afirst-person perspective was recognized (YES in S116), in step S120, theperspective switching control unit 15 switches from a third-personperspective to a first-person perspective.

Then, in step S124, the playback control unit 16 performs pan/zoom-ineffect processing based on the switch of perspective by the perspectiveswitching control unit 15. In the example illustrated in FIG. 10, asindicated as section 201, the playback control unit 16 successivelyswitches the content to be played back from the content 101B to content102B and then 103B. At this stage, in order to make up for thedifference in perspectives among the respective pieces of content, theplayback control unit 16 plays back a morphing image that gradually pansand zooms in. Consequently, the user 2 can experience a feeling ofcoming closer to the actor 4 while gradually moving around behind theactor 4.

Next, in step S128, the playback control unit 16 performs dolly-ineffect processing. In the example illustrated in FIG. 10, as indicatedas section 202, the playback control unit 16 successively switches thecontent to be played back from the content 103B to content 104, 105,106, and then 107. At this stage, in order to make up for the differencein perspectives among the respective pieces of content, the playbackcontrol unit 16 plays back a morphing image that gradually dollies in.Consequently, the user 2 can enjoy the content as if he/she were playingthe role of the actor 4 while moving closer step-by-step to the actress3.

Next, as illustrated in FIG. 9, in step S204, the editing unit 12performs editing processing on the content acquired by the contentacquisition unit 11 in a first-person perspective. For example, althoughnot illustrated in FIG. 10, the editing unit 12 replaces the face imageof the actress 3 appearing in the content 107 in a first-personperspective with the face image of another person.

Then, in step S208, the playback control unit 16 controls so that thecontent edited by the editing unit 12 is displayed by the display unit60 and the audio output unit 70.

Next, in steps S212 to S216, the control unit 10 operates in the samemanner as described above in steps S112 to S116.

If an instruction to switch from a first-person perspective to athird-person perspective was recognized (YES in S216), in step S220, theperspective switching control unit 15 switches from a first-personperspective to a third-person perspective.

Then, in step S224, the playback control unit 16 performs dolly-backeffect processing based on the switch of perspective by the perspectiveswitching control unit 15. In the example illustrated in FIG. 10, asindicated as section 203, the playback control unit 16 successivelyswitches the content to be played back from the content 107 to content106, 105, and then 104. At this stage, in order to make up for thedifference in perspectives among the respective pieces of content, theplayback control unit 16 plays back a morphing image that graduallydollies back. Consequently, the user 2 can enjoy the content as ifhe/she were playing the role of the actor 4 while moving step-by-stepaway from the actress 3.

Next, in step S228, the playback control unit 16 performs pan/zoom-outeffect processing. In the example illustrated in FIG. 10, as indicatedas section 204, the playback control unit 16 successively switches thecontent to be played back from the content 104 to content 103B, 102B,and then 101B. At this stage, in order to make up for the difference inperspectives among the respective pieces of content, the playbackcontrol unit 16 plays back a morphing image that gradually pans/zoomsout. Consequently, the user 2 can experience a feeling of graduallymoving away from and around behind the actor 4 to a position overlookingthe positional relationship between the actress 3 and the actor 4.

Then, the processing returns to step S104 illustrated in FIG. 8.

In the above, the operation processing of the HMD 1 according to a firstembodiment of the present disclosure was described.

2-1-4. First Modified Example of the Present Disclosure

The present modified example is an embodiment in which the editing unit12 replaces the face image of the imaging target registered in thecontent with the face image of the user 2, and replaces the voice ofthat imaging target with the voice of the user 2. The operationprocessing according to the present modified example will now bedescribed with reference to FIG. 11.

FIG. 11 is a diagram illustrating operation processing of the HMD 1according to the first modified example of the present disclosure. Asillustrated in FIG. 11, the playback control unit 16 plays back contentin which the face image of the actor 4 has been replaced with the faceimage of the user 2, and superimposes over that a subtitle which showsthe lines spoken by the actress 3 and the actor 4 in that scene. Byspeaking the lines spoken by the actor 4 in subtitle 5, the user 2 canenjoy the content as if he/she is playing the role of the actor 4. Atthis stage, the playback control unit 16 temporarily stops the playbackof the content on which the subtitle 5 has been displayed, and when ithas been recognized by the recognition unit 13 that the user 2 hasspoken all of the lines of the actor 4 shown in the subtitle 5, theplayback control unit 16 may end the temporary stoppage and play backthe next scene.

It is noted that the playback control unit 16 can proceed to the nextscene if lip movement is recognized instead of voice recognition. As aresult, the user can keep playing back the content by reading thesubtitle 5 even in such a noisy environment that voice recognition isdifficult to perform. To recognize such lip movement, the playbackcontrol unit 16 can use, for example, the technique disclosed in “LipMovement Reconition Using Template Matching” by Kiyoshi Takita, TakeshiNagayasu, Hidetsugu Asano, Kenji Terabayashi, and Kazunori Umeda, TheJapan Society of Mechanical Engineers, Robotics and MecatronicsPresentation 2011, Collection of Presentation Materials 2P1 to Q12, May2011. This technique recognizes lip movement based on template matchingby using a low-resolution image obtained from an image of the mouth, theshape of the lips, and the shape of the open portion that is formed whenthe mouth is opened as feature quantities.

Further, the HMD 1 can also be enjoyed by a plurality of users bysharing content edited by the editing unit 12, namely, content in whicha part of the content has been replaced with the user 2, over theInternet.

In the above, the first modified example of the present disclosure wasdescribed.

2-2. Second Embodiment of the Present Disclosure

The present embodiment is an embodiment in which the perspective whenplaying back movie content is switched to at least either a first-personperspective or a third-person perspective based on position information.Although the first embodiment according to the present disclosure isrealized by the HMD 1 described above with reference to FIG. 4, thepresent embodiment is realized by a smartphone 6 having the sameconfiguration as the HMD 1. An outline of the present embodiment willnow be described with reference to FIG. 12.

2-2-1. Outline

FIG. 12 is a diagram illustrating an outline of the smartphone 6according to the second embodiment of the present disclosure. Thesmartphone 6 is gripped by the user 2, and displays content 101B to theuser 2. The smartphone 6 includes the imaging unit 20 and the audioinput unit 30 on a screen provided in the display unit 60. A part ofcontent 101D is replaced in real time with the face image and voice ofthe user 2 acquired in real time.

The perspective switching control unit 15 according to the presentembodiment switches the perspective for playing back the content basedon the position of the smartphone 6. More specifically, the perspectiveswitching control unit 15 switches the perspective when playing backcontent to the perspective from the position corresponding to theposition information acquired by the position information acquisitionunit 14 in the space of the content. Here, the position corresponding tothe position information acquired by the position informationacquisition unit 14 in the space of the content is the position wherethe smartphone 6 is currently present in the imaging location where thatcontent was captured. The switching of the perspective based on theposition of the smartphone 6 that is performed by the perspectiveswitching control unit 15 will now be described.

As described regarding the HMD 1 with reference to FIG. 7, thesmartphone 6 acquires and plays back with the content acquisition unit11 the content corresponding to the position information acquired by theposition information acquisition unit 14. At this stage, for example, achange can arise in the position information acquired by the positioninformation acquisition unit 14 due to the smartphone 6 having beenmoved while still held by the user 2, or the bearing and elevation withwhich the user 2 is holding the smartphone 6 having changed. In such acase, the content acquisition unit again acquires the contentcorresponding to the changed position information, and the playbackcontrol unit 16 switches the content to be played back to there-acquired content.

Consequently, when the user 2 moves through a real space while holdingthe smartphone 6, the perspective of the content to be played back onthe display unit 60 switches accordingly. For example, in the exampleillustrated in FIG. 1, if the user 2 moves to the position ofperspective 2-2 in the real world, the perspective switching controlunit 15 switches the perspective to perspective 2-2 so that the content102 is played back. Further, if the user 2 moves to the position ofperspective 2-3 in the real world, the perspective switching controlunit 15 switches the perspective to perspective 2-3 so that the content103 is played back.

It is noted that although the position corresponding to the positioninformation acquired by the position information acquisition unit 14 inthe space of the content was described as a position in which thesmartphone 6 is currently present in the imaging location where thatcontent was captured, the present embodiment is not limited to thatexample. For example, as the position corresponding to the positioninformation acquired by the position information acquisition unit 14 inthe space of the content, a relative position with respect to a specificposition may be employed. Below, an example will now be described inwhich the perspective is switched based on a relative position.

For example, when using the smartphone 6 indoors, a specific indoorposition is set as a position corresponding to the location where theimaging target was in the imaging location, namely, a positioncorresponding to an origin where the X-axis, Y-axis, and Z-axis in thescene 100 illustrated in FIG. 1 intersect. Further, the perspectiveswitching control unit 15 switches the perspective based on a relativechange in the position/bearing/elevation of the smartphone 6 withrespect to the set specific position. For example, if the smartphone 6has come closer to that specific position, the perspective switchingcontrol unit 15 switches the perspective so as to be closer to theimaging target. Further, if the smartphone 6 has moved further away fromthat specific position, the perspective switching control unit 15switches the perspective so as to be further away from the imagingtarget.

Generally, the location where movies and television dramas are filmed isoften far away from where people live, such as overseas or at a touristsite. Therefore, by setting a specific position and changing theperspective when playing back content based on the relative positionwith respect to that specific position, the user 2 can enjoy content byeasily using the smartphone 6 in his/her own room, at school, at a stafflounge at work and the like, without having to go to the filminglocation.

In the above, an outline of the smartphone 6 according to the secondembodiment of the present disclosure was described. Next, the operationprocessing of the smartphone 6 according to the present embodiment willbe described. It is noted that the operation processing of the HMD 1described above with reference to FIG. 7 is performed in the same mannerby the smartphone 6 too. However, the smartphone 6 according to thepresent embodiment performs the content display processing according tostep S40-2 illustrated in FIGS. 13 and 14 as the content displayprocessing performed in step S40 illustrated in FIG. 7. Accordingly, inthe below, the content display processing performed in step S40-2, whichis a feature of the operation processing performed by the smartphone 6according to the present embodiment, will be described with reference toFIGS. 13 and 14.

2-2-2. Operation Processing

FIGS. 13 and 14 are flowcharts illustrating the operation processingperformed by the control unit 10 according to the second embodiment ofthe present disclosure. As illustrated in FIG. 13, first, in step S302,the perspective switching control unit 15 determines whether theperspective is a third-person perspective.

If the perspective is not a third-person perspective (NO in S302), theprocessing proceeds to the below-described step S404 illustrated in FIG.14.

On the other hand, if the perspective is a third-person perspective (YESin S302), in steps S304 to S312, the control unit 10 performs the sameprocessing as that in the above-described steps S104 to S112).

Next, in step S316, the position information acquisition unit 14acquires position information in the same manner as in step S20illustrated in FIG. 7.

Then, in step S318, the perspective switching control unit 15 determineswhether there is a change in the position information acquired by theposition information acquisition unit 14. For example, the perspectiveswitching control unit 15 determines that there is a change in theposition information if the position information acquired by theposition information acquisition unit 14 has changed from the positioninformation acquired the previous time.

If there is no change in the position information (NO in S318), theprocessing returns to step S304.

On the other hand, if there is a change in the position information (YESin S318), in step S320, the perspective switching control unit 15switches the perspective to the perspective from the positioncorresponding to the position information acquired by the positioninformation acquisition unit 14.

Next, in step S324, based on the position information acquired by theposition information acquisition unit 14, the perspective switchingcontrol unit 15 determines whether the position of the smartphone 6 iscloser to the imaging target than the previous time. For example, theperspective switching control unit 15 determines that the position ofthe smartphone 6 is closer to the imaging target if the user 2 holdingthe smartphone 6 is closer to the origin illustrated in FIG. 1.Additionally, the perspective switching control unit 15 also determinesthat the position of the smartphone 6 is closer to the imaging target ifthe user 2 holding the smartphone 6 is closer to the specific positionthat was pre-set indoors.

If the position of the smartphone 6 is closer to the imaging target (YESin S324), in step S328, the playback control unit 16 plays back amorphing image that gradually pans and zooms in based on the switch inperspective carried out by the perspective switching control unit 15.

On the other hand, if the position of the smartphone 6 is not closer tothe imaging target (NO in S324), in step S332, the playback control unit16 plays back a morphing image that pans and zooms out based on theswitch in perspective carried out by the perspective switching controlunit 15.

Next, in step S336, the perspective switching control unit 15 determineswhether the position of the smartphone 6 is in range for performing thedisplay in a third-person perspective (is beyond the range correspondingto the imaging target) based on the position information acquired by theposition information acquisition unit 14. For example, the perspectiveswitching control unit 15 determines that the position of the smartphone6 is in range for performing the display in a third-person perspectiveif the position of the smartphone 6 and the position of the imagingtarget are further away from each other than a predetermined distance.Additionally, the perspective switching control unit 15 can alsodetermine that the position of the smartphone 6 is in range forperforming the display in a third-person perspective if the smartphone 6is positioned to the side or in front of the imaging target (at aposition where the actor is seen from in front or the side).

If the smartphone 6 is in range for performing the display in athird-person perspective (YES in S336), the processing returns to stepS304. Thus, if the position of the smartphone 6 is in range forperforming the display in a third-person perspective, in the above stepsS324, S328, and S332, the playback control unit 16 plays back a morphingimage that pans and zooms in/out. Consequently, the playback controlunit 16 can play back content in a third-person perspective while givingthe user 2 an impression as if the images were captured while the cameraposition is moving.

On the other hand, if the smartphone 6 is not in range for performingthe display in a third-person perspective (NO in S336), in step S340,the playback control unit 16 performs pan/dolly-in effect processing.Although the smartphone 6 plays back the content in the subsequentprocessing in a first-person perspective, by performing pan/dolly-ineffect processing, the playback of the content in a third-personperspective and the subsequent playback of the content in a first-personperspective can be smoothly linked together.

Next, as illustrated in FIG. 14, in steps S304 to S312, the control unit10 performs processing in the same manner as in the above-describedsteps S204 to S212.

Then, in steps S416 to S424, the position information acquisition unit14 performs processing in the same manner as in the above-describedsteps S316 to S324.

If the position of the smartphone 6 is closer to the imaging target (YESin S424), in step S428, the playback control unit 16 plays back amorphing image that gradually pans and dollies in based on the switch inperspective carried out by the perspective switching control unit 15.

On the other hand, if the position of the smartphone 6 is not closer tothe imaging target (NO in S424), in step S432, the playback control unit16 plays back a morphing image that pans and dollies back based on theswitch in perspective carried out by the perspective switching controlunit 15.

Next, in step S436, the perspective switching control unit 15 determineswhether the position of the smartphone 6 is in range for performing thedisplay in a first-person perspective (is in the range corresponding tothe imaging target) based on the position information acquired by theposition information acquisition unit 14. For example, the perspectiveswitching control unit 15 determines that the position of the smartphone6 is in range for performing the display in a first-person perspectiveif the position of the smartphone 6 and the position of the imagingtarget are closer to each other than a predetermined distance.Additionally, the perspective switching control unit 15 can alsodetermine that the position of the smartphone 6 is in range forperforming the display in a first-person perspective if the smartphone 6is positioned to the rear of the imaging target (at a position where theactor is seen from behind).

If the smartphone 6 is in range for performing the display in afirst-person perspective (YES in S436), the processing returns to stepS404. Thus, if the position of the smartphone 6 is in range forperforming the display in a first-person perspective, in the above stepsS424, S428, and S332, the playback control unit 16 plays back a morphingimage that pans and dollies in/out. Consequently, the playback controlunit 16 can play back content in a first-person perspective while givingthe user 2 the feeling that he/she is freely moving through the contentspace playing the role of an actor.

On the other hand, if the smartphone 6 is not in range for performingthe display in a first-person perspective (NO in S436), in step S440,the playback control unit 16 performs pan/zoom-out effect processing.Although the smartphone 6 plays back the content in the subsequentprocessing in a third-person perspective, by performing pan/zoom-outeffect processing, the playback of the content in a first-personperspective and the subsequent playback of the content in a third-personperspective can be smoothly linked together.

The processing then returns to step S304 illustrated in FIG. 13.

In the above, the operation processing of the smartphone 6 according tothe present embodiment was described.

2-2-3. Second Modified Example of the Present Disclosure

The present modified example is an embodiment in which, when a pluralityof smartphones 6 are being used simultaneously, a part of the contentdisplayed by each smartphone 6 is replaced with the profile of the ownerof another smartphone 6. Below, operation of the smartphone 6 accordingto the present modified example will be described with reference to FIG.15.

FIG. 15 is a diagram illustrating operation of the smartphone 6according to the second modified example of the present disclosure. Asillustrated in FIG. 15, a user 2A and a user 2B are facing each other.The user 2A and the user 2B are holding a smartphone 6A and a smartphone6B, respectively, which are pointed at each other. At this stage, thesmartphone 6A held by the user 2A plays back content in which the faceimage of the actress 3 appearing in content 131 has been replaced withthe face image of the user 2B captured by the imaging unit 20 (notillustrated) provided on the rear face of the smartphone 6A. Similarly,the smartphone 6B held by the user 2B plays back content in which theface image of the actor 4 appearing in content 132 has been replacedwith the face image of the user 2A captured by the imaging unit 20 (notillustrated) provided on the rear face of the smartphone 6B.

It is noted that the smartphones 6 may exchange with each other the faceimage of the user 2 who is holding the smartphone 6 that was captured byan imaging unit 20 provided on the same face as the display unit 60. Forexample, in the example illustrated in FIG. 15, the 6A may send to thesmartphone 6B the face image of the user 2A captured by an imaging lens20A that is provided on the same face as a display unit 60A. Thisenables content to be played back on the smartphone 6B in which the faceimage of the actor 4 appearing in content 132 has been replaced with theface image of the user 2A received from the smartphone 6A.

Thus, when a plurality of smartphones 6 are simultaneously used by aplurality of users 2, a part of the content to be played back can bereplaced with the profile of another user 2. Consequently, the samecontent can be enjoyed by a plurality of users 2 together.

3. CONCLUSION

As described above, the information processing apparatus according tothe embodiments of the present disclosure can switch perspective whenplaying back content to at least either a first-person perspective or athird-person perspective, and play back that content. Further, theinformation processing apparatus according to an embodiment of thepresent disclosure can not only switch perspective, it can also edit sothat a part of the content is replaced with the user 2. Consequently,the user 2 can enjoy the content as if he/she were actually playing therole of an actor or a character appearing in the content.

Further, the information processing apparatus according to an embodimentof the present disclosure can also switch the perspective in whichcontent is played back based on the position, bearing, or elevation ofthe information processing apparatus. This allows the user 2 toenjoyably watch the content from a given perspective.

It should be understood by those skilled in the art that variousmodifications, combinations, sub-combinations and alterations may occurdepending on design requirements and other factors insofar as they arewithin the scope of the appended claims or the equivalents thereof.

For example, in the above-described embodiments of the presentdisclosure, although the HMD 1 or the smartphone 6 were described asapplied examples, the present technology is not limited to thoseexamples. For example, the above-described technology can also beapplied in an information processing apparatus that is used for karaoke.Recently, some karaoke establishments offer a service that when the user2 sings a song, a promotional video or a live concert in which theartist who sings that song appears is played back. By replacing the faceimage of the artist appearing in the promotional video or live concertwith the face image of the user 2 using the above-described technologyin an information processing apparatus that provides such a karaokeservice, the user 2 can enjoy karaoke as if he/she were that artist.Moreover, in a song that is sung by a plurality of people, such as aduet, by replacing the face images of the plurality of artists, karaokecan be enjoyed by making a plurality of people become those artists.

Further, the hardware such as the CPU, ROM, and RAM in the informationprocessing apparatus can also create a computer program that realizesthe same functions as each constituent unit in the above-described HMD 1or smartphone 6. In addition, according to an embodiment of the presentdisclosure, a storage medium on which that computer program is recordedis also provided.

Additionally, the present technology may also be configured as below.

(1) An information processing apparatus including:

a perspective switching control unit configured to switch a perspectivewhen playing back content acquired by a content acquisition unit to atleast one of a first-person perspective and a third-person perspective;

an editing unit configured to edit a part of the content; and

a playback control unit configured to play back the content edited bythe editing unit in the at least one of the first-person perspective andthe third-person perspective to which the perspective has been switchedby the perspective switching control unit.

(2) The information processing apparatus according to (1), wherein thefirst-person perspective is a perspective of an object that appears inthe content.

(3) The information processing apparatus according to (2),

wherein the content includes still image-based or moving image-basedimage data, and

wherein the object is a living being, a virtual living being, or acharacter.

(4) The information processing apparatus according to (2) or (3),

wherein the editing unit is configured to replace image data of theobject appearing in the content with image data of another objectappearing in other content acquired by the content acquisition unit.

(5) The information processing apparatus according to any one of (2) to(4),

wherein the content includes audio data, and

wherein the editing unit is configured to replace audio data of theobject appearing in the content with audio data of another objectappearing in other content acquired by the content acquisition unit.

(6) The information processing apparatus according to any one of (1) to(5), further including:

a recognition unit configured to recognize a user gesture,

wherein the perspective switching control unit is configured to switchthe perspective based on the user gesture recognized by the recognitionunit.

(7) The information processing apparatus according to any one of (1) to(6), wherein the playback control unit is configured to, when theperspective is switched by the perspective switching control unit, playback a morphing image that gradually changes from the perspective beforethe perspective has been switched to the switched perspective.

(8) The information processing apparatus according to (7), wherein theplayback control unit is configured to, when the perspective is switchedby the perspective switching control unit from the first-personperspective to the third-person perspective, play back the morphingimage in a manner that the perspective first dollies back, and thenzooms out.

(9) The information processing apparatus according to (7) or (8),wherein the playback control unit is configured to, when the perspectiveis switched by the perspective switching control unit from athird-person perspective to a first-person perspective, play back themorphing image in a manner that the perspective first zooms in, and thendollies in.

(10) The information processing apparatus according to any one of (7) to(9), further including:

a position information acquisition unit configured to acquire positioninformation about the information processing apparatus,

wherein the perspective includes a perspective from a given position ina space of the content, and

wherein the perspective switching control unit is configured to switchto the perspective to a perspective from a position corresponding to theposition information acquired by the position information acquisitionunit in the space of the content.

(11) The information processing apparatus according to (10), wherein theplayback control unit is configured to, when the perspective is switchedby the perspective switching control unit and the perspective ispositioned within an area corresponding to an object in the space of thecontent, play back a morphing image in a manner that the perspectivedollies in or dollies back.

(12) The information processing apparatus according to (10) or (11),wherein the playback control unit is configured to, when the perspectiveis switched by the perspective switching control unit and theperspective is positioned out of an area corresponding to an object inthe content space, play back a morphing image in a manner that theperspective zooms in or zooms out.

(13) The information processing apparatus according to any one of (10)to (12), wherein the position information is measured using GPS.

(14) The information processing apparatus according to any one of (10)to (12), wherein the position information is measured based on arelative position with respect to a specific position.

(15) The information processing apparatus according to any one of (1) to(14), wherein the playback control unit is configured to, when theperspective is switched by the perspective switching control unit, playback the content in a manner that a localized position of an audio imagegradually changes.

(16) The information processing apparatus according to any one of (1) to(15), wherein the playback control unit is configured to play backpredetermined audio data when the perspective is switched by theperspective switching control unit.

(17) A non-transitory computer-readable recording medium having aprogram recorded thereon, the program causing a computer to execute:

switching a perspective when playing back content acquired by a contentacquisition unit to at least one of a first-person perspective and athird-person perspective;

editing a part of the content based on the switching of the perspective;and

playing back the edited content in the at least one of the first-personperspective and the third-person perspective to which the switchedperspective has been switched.

What is claimed is:
 1. An information processing apparatus comprising: aperspective switching control unit configured to switch a perspectivewhen playing back content acquired by a content acquisition unit to atleast one of a first-person perspective and a third-person perspective;an editing unit configured to edit a part of the content; and a playbackcontrol unit configured to play back the content edited by the editingunit in the at least one of the first-person perspective and thethird-person perspective to which the perspective has been switched bythe perspective switching control unit.
 2. The information processingapparatus according to claim 1, wherein the first-person perspective isa perspective of an object that appears in the content.
 3. Theinformation processing apparatus according to claim 2, wherein thecontent includes still image-based or moving image-based image data, andwherein the object is a living being, a virtual living being, or acharacter.
 4. The information processing apparatus according to claim 2,wherein the editing unit is configured to replace image data of theobject appearing in the content with image data of another objectappearing in other content acquired by the content acquisition unit. 5.The information processing apparatus according to claim 2, wherein thecontent includes audio data, and wherein the editing unit is configuredto replace audio data of the object appearing in the content with audiodata of another object appearing in other content acquired by thecontent acquisition unit.
 6. The information processing apparatusaccording to claim 1, further comprising: a recognition unit configuredto recognize a user gesture, wherein the perspective switching controlunit is configured to switch the perspective based on the user gesturerecognized by the recognition unit.
 7. The information processingapparatus according to claim 1, wherein the playback control unit isconfigured to, when the perspective is switched by the perspectiveswitching control unit, play back a morphing image that graduallychanges from the perspective before the perspective has been switched tothe switched perspective.
 8. The information processing apparatusaccording to claim 7, wherein the playback control unit is configuredto, when the perspective is switched by the perspective switchingcontrol unit from the first-person perspective to the third-personperspective, play back the morphing image in a manner that theperspective first dollies back, and then zooms out.
 9. The informationprocessing apparatus according to claim 7, wherein the playback controlunit is configured to, when the perspective is switched by theperspective switching control unit from a third-person perspective to afirst-person perspective, play back the morphing image in a manner thatthe perspective first zooms in, and then dollies in.
 10. The informationprocessing apparatus according to claim 7, further comprising: aposition information acquisition unit configured to acquire positioninformation about the information processing apparatus, wherein theperspective includes a perspective from a given position in a space ofthe content, and wherein the perspective switching control unit isconfigured to switch to the perspective to a perspective from a positioncorresponding to the position information acquired by the positioninformation acquisition unit in the space of the content.
 11. Theinformation processing apparatus according to claim 10, wherein theplayback control unit is configured to, when the perspective is switchedby the perspective switching control unit and the perspective ispositioned within an area corresponding to an object in the space of thecontent, play back a morphing image in a manner that the perspectivedollies in or dollies back.
 12. The information processing apparatusaccording to claim 10, wherein the playback control unit is configuredto, when the perspective is switched by the perspective switchingcontrol unit and the perspective is positioned out of an areacorresponding to an object in the content space, play back a morphingimage in a manner that the perspective zooms in or zooms out.
 13. Theinformation processing apparatus according to claim 10, wherein theposition information is measured using GPS.
 14. The informationprocessing apparatus according to claim 10, wherein the positioninformation is measured based on a relative position with respect to aspecific position.
 15. The information processing apparatus according toclaim 1, wherein the playback control unit is configured to, when theperspective is switched by the perspective switching control unit, playback the content in a manner that a localized position of an audio imagegradually changes.
 16. The information processing apparatus according toclaim 1, wherein the playback control unit is configured to play backpredetermined audio data when the perspective is switched by theperspective switching control unit.
 17. A non-transitorycomputer-readable recording medium having a program recorded thereon,the program causing a computer to execute: switching a perspective whenplaying back content acquired by a content acquisition unit to at leastone of a first-person perspective and a third-person perspective;editing a part of the content based on the switching of the perspective;and playing back the edited content in the at least one of thefirst-person perspective and the third-person perspective to which theswitched perspective has been switched.